Papers with vanilla transformer
Multi-split Reversible Transformers Can Enhance Neural Machine Translation (2021.eacl-main)
Copied to clipboard
| Challenge: | Large-scale transformers have been shown to improve neural machine translation performance but training these wider and deeper networks could be extremely memory intensive. |
| Approach: | They propose a multi-split based reversible transformer and a backpropagation algorithm that does not need to store activations for most layers. |
| Outcome: | The proposed model outperforms the vanilla transformer by at least 1.4 BLEU points in three datasets. |
Linearizing Transformer with Key-Value Memory (2022.emnlp-main)
Copied to clipboard
| Challenge: | Efficient transformer variants with linear time complexity have been developed to mitigate the quadratic computational overhead of the vanilla transformer. |
| Approach: | They propose a linear time complexity transformer variant that reduces the quadratic computational overhead of the vanilla transformer by using a recurrent-style incremental computation similar to kernel-based transformers. |
| Outcome: | The proposed method reduces the performance gap while achieving the same efficiency even with short generation. |
Prior Knowledge and Memory Enriched Transformer for Sign Language Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for sign language translation do not explore all of them . visual and textual understanding and additional prior knowledge learning are challenging . |
| Approach: | They propose a method which integrates auxiliary information into vanilla transformer for SLT . they use visual-textual context information and additional auxiliary knowledge of a word . |
| Outcome: | The proposed method improves the understanding of sign language videos with visual and textual understanding and additional prior knowledge learning. |
∞-former: Infinite Memory Transformer (2022.acl-long)
Copied to clipboard
| Challenge: | Several efficient transformers have been proposed, but they all have a finite memory capacity and are forced to drop old information. |
| Approach: | They propose an unbounded long-term memory extension that extends the vanilla transformer by using a continuous-space attention mechanism to attend over the long-time memory. |
| Outcome: | The proposed model can model arbitrarily long contexts while keeping the computation budget fixed. |
The Devil in Linear Transformer (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing linear transformers suffer from performance degradations on various tasks and corpus. |
| Approach: | They propose a new linear attention that replaces scaling with a normalization to stabilize gradients and confine attention to neighbouring tokens in early layers. |
| Outcome: | The proposed model outperforms vanilla transformers on the long-range arena benchmark while being significantly more space-time efficient. |
Prediction Difference Regularization against Perturbation for Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for regularizing input perturbation are limited by under-fitting of training data. |
| Approach: | They propose a method that can reduce over-fitting and under-fitting at the same time. |
| Outcome: | The proposed method can reduce over-fitting and under-fitturing while making the model less sensitive to small input changes and more robust to under-perturbed training data. |